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Overview 


■ The current status of Exploration Medical Capability 
(ExMC) Gap 3.01 

■ Updated assessment and analysis of the training 
gaps 

■ Validation of training programs 

■ Metrics for quantification of training effectiveness 

■ What else do we know? 

■ Lessons learned so far 

■ Areas for future investigation 




ExMCGap 3.01 


■ “We do not know the optimal training 
methods for in-flight medical conditions 
identified on the Space Medicine 
Exploration Medical Condition List 
[SMEMCL] taking into account the Crew 
Medical Officer's clinical background." 






ExMC Gap 3.01 - Why is it 
important? 


■ Majority of historical in-flight medical problems 
were minor 

■ Though all crews undergo training in more 
advanced emergency medical procedures, these 
skills remain untested in a real-life incident 

■ e.g., Advanced cardiac life support (ACLS) 

■ How can we be sure our training procedures are 
effective? 


2. 


Hamilton D, Smart K, Melton S, Polk JD, Johnson-Throop K. Autonomous medical care for exploration class space missions. J Trauma. 2008; 64: S354-63.) 
"ExMC Gap Report 3.01," NASA Human Research Wiki. Retrieved from https://humanresearchwiki.jsc.nasa. gov/index. php?title=3. 01. 




The need for medical preparation 


■ “Historically, the difficulties associated with illness 
and injury have accounted for the failures of more 
expeditions on Earth's frontiers than any other 
single technical or environmental reason." 

■ It is only a matter of time before there is a more 

severe medical emergency in space 
■ Integrated Medical Model (IMM) helps predict incidence 
rates and incidence proportions 


Summers RL, Johnston SL, MarshburnTH, Williams DR. Emergencies in space. Ann Emerg Med. 2005;46:177-84. 




A paradigm shift 


■ Exploration missions will require a paradigm 
shift in medical preparation due to new 
limitations and unknown risks 

■ Current mission planning: 

■ Real-time ground support 

■ Evacuation to Earth contingencies 

■ Future mission planning: 

■ Communication delay 

■ Need to treat to definitive resolution or stabilization 




Current gap approach 


■ Can we design a theoretical way to measure 
training effectiveness? 

■ Targets: 

■ Identify a necessary skill-set based on SMEMCL 
■ Determine the best training method for each skill 

■ Validate current and proposed training methods 
for both crew and ground support (flight surgeons 
and biomedical engineers [BMEs]) 




Current gap approach 


■ Develop a metric to quantify the effectiveness 
of training methods 

■ Integrate lessons learned from rigorous testing 
into exploration training methodology 




Gap 3.01 -The Report 


■ Outlines current training flows for astronaut 
candidates, crew medical officer (CMO), surgeon, and 
BME 

■ Identifies limitations imposed on training algorithms 
by exploration missions 

■ Lack of knowledge about medical conditions specific to 
spaceflight 

■ Microgravity 

■ Problems pertaining to knowledge and skill retention 

■ Current training methods are not adequate for 
exploration missions 


"ExMC Gap Report 3.01," NASA Human Research Wiki. Retrieved from 
https://humanresearchwiki.jsc.nasa. gov/index. php?title=3.oi. 




Aiming for perfection 


■ The best possible medical training for crew 

■ The best possible preparation for ground 
support 

■ The best medical technology and equipment 

■ In-flight medical autonomy (ideally) 




Where are we going, and how do 
we get there? 







A new approach to the old gap 


1. Existing gap report reviewed 

2. Limited literature search performed 

3. Areas of training defined 

4. Extrapolation of a comprehensive list of 
questions regarding training 

5. Identification of subject matter experts 
(SME) in training 

6 . Initial SME interviews to further improve 
questionnaire 




Training gaps questionnaire 


■ Who better to help answer questions about 
training than experts in training? 

■ What would we ask those experts...? 

■ In general, what are the tried and true best 
methods of medical education? 

■ How should we teach each specific piece of 
medical knowledge or individual skill? 




Training gaps questionnaire 


OPTIMIZATION OF TRAINING: STRATEGIZING CLOSURE OF GAP 3.01 


Validation 

• How are education and training validated in general? 

• How are medical education and medical training validated? 

o How are medical objective standardized clinical examinations (OSCEs) designed and 
validated? 

o How are interactive case-based simulations, educational tools, and assessments created 
and validated (e.g., United States Medical Licensing Exams)? 

• How can we use the standards established in the validation of medical education teaching and 
evaluation methods to validate training methods for exploration class missions? 

• How can we validate training methods priorto implementation of a training program for an 
exploration class mission? 

M e tries /Qu an tifica tion 

• What metric do we use to assess the effectiveness of our training programs, strategies, and 
techniques? 

• How do outside groups (e.g., EMTs, military) certify CREW and GROUND SUPPORT? 

• How do we design studies to test the following: 

o The besttraining methods and techniques for a defined skill/topic? 
o The effectiveness of our tra ining progra ms? 
o The metrics with which we evaluate our training programs? 




Training gaps questionnaire 


Training Methods -Telemedicine 

• What a re the gold sta nda rcls f or tra ining users of telemedicine in a terrestria I environment? 

• How can we adaptthe current terrestrial gold standards to the limitations of ourtraining 
environment? 

Training Methods -Just-in-Time Training 

• What are the gold standards for just-in-time training programs in a terrestrial environment? 

• How can we adaptthe current terrestrial gold standards to the limitations of ourtraining 
environment? 

Tra in in g Methods - Dental 

• What: are the gold standards for dental-focused training in a terrestrial environment? 

• How can we adaptthe current terrestrial gold standards to the limitations of ourtraining 
environment? 

Training Methods - Medical Knowledge 

• What are the gold standards for medical knowledge training in a terrestrial environment? 

• How can we adaptthe current terrestrial gold standards to the limitations of ourtraining 
environment? 

Tra in in g Meth ods -Me dical Skills/Pro ce d ures 

• What are the gold standards for training in medical skills/procedures in a terrestrial 
environment? 




Locating subject matter experts 







Locating subject matter experts 


■ Internal organization (NASA and Wyle) 

■ Current training leads and those already directly 
involved with training 

■ External experts 

■ Previous participants in summits on training (e.g., 
dental) 

■ SciVal Expert Locator ® (aka Collexis) 

■ Literature search 

■ Resulted in a spreadsheet of contacts 
organized by area of expertise 




Improving the questionnaire 


■ Initial meetings conducted with internal 
training experts for feedback on the 
questionnaire 

■ Some modifications made to wording and 
specificity of questions 

■ General consensus... a potentially useful approach 




Addressing the questions... 






Initial research: Validation and 
metrics 


VALIDITY 

Does the test or measurement actually quantify 
the attribute it is intended to quantify? 

"...a test is valid if it measures what it purports 

to measure..." (Kelly, 1927) 


Colliver JA, Conlee MJ, Verhulst SJ. From test validity to construct validity... and back? Med Educ. 2012; 46: 366-371. 




Validity vs. validation 


■ Validity is a property of a attribute 

■ Similar to the concept of truth 

■ i.e., Are our methods of training valid? 

■ Validation is a process of theory testing to 
determine if an attribute possesses the 
characteristic of being valid 

■ i.e., How can we test our methods of training to 
determine their validity? 


Borsdoom D, Mellenbergh GJ. The concept of validity. Psychol Rev. 2004; 111: 1061-1071. 




Reliability 


■ The consistency of measurement outcomes 
over multiple iterations of a test 

■ Quantifies the degree of variation in scores to 
determine if the test is reproducible 

■ An assessment of the amount of random error 

■ Necessary but not sufficient for establishing 
validity 


Downing SM. Reliability: on the reproducibility of assessment data. MedEduc. 2004;38:1006-1012. 




How can we validate or measure 
effectiveness of training methods ? 


■ Types of evaluations used in medical education 

Written assessment (knowledge/cognitive ability) 

■ Rating of directly observed performance 

■ Standardized performance-based evaluation 

■ Observed standardized clinical encounters (OSCEs) 

■ Simulated or standardized patients 

■ Computer-based simulations 

■ Other methods of evaluation 

■ Review of written documentation (e.g., notes, orders) 

■ Other statistics (e.g., complications, mortality, awards) 

1. Downing SM. Reliability: on the reproducibility of assessment data. MedEduc. 2004;38:1006-1012. 

2. Kreiter CD, Bergus G. The validity of performance-based measures of clinical reasoning and alternative 
approaches. MedEduc. 2009;43:320-325. 




Knowledge assessments 


■ Written test format (e.g., multiple choice 

questions) 

■ One of the simplest of the medical evaluation 
tools 

■ Relatively straightforward to establish moderate 
reliability and validity 

■ Limited applications beyond basic assessment of 
knowledge acquisition and retention 


1 . 


2 . 


CrossleyJ, Humphries G, Jolly B. Assessing health professionals. MedEduc. 2002;36:800-80 4. 
Downing SM. Reliability: on the reproducibility of assessment data. MedEduc. 2004;38:1006-1012. 




Rating of directly observed 
performance 


■ An expert evaluation of clinical performance in a rea 
world setting (e.g., on the wards) 

■ Oral examinations attempt to mimic clinical context 

■ Used to assess the following: 

■ How the subject integrates knowledge and skill into 
behavior 

■ Clinical judgment 

■ Professional competencies 

■ Outcomes are expert ratings the performance of 
subject 

■ Point scale (e.g., 1-5) 

Dichotomous checklist 




Rating of directly observed 
performance 


■ Limitations 

■ Competencies and behaviors are difficult to define 
and not easily measured 

■ High degree of variability 

■ Time intensive 

■ Reliability and validity evidence in medical 
education literature moderate, but format 
widely used 

Lurie SJ, Mooney CJ, LynessJM. Commentary: pitfalls in assessment of competency-based educational objectives. Acad 
Med. 2011; 86: 412-414. 

Tabuenca A, Welling R, Sachdeva AK, Blair PG, Horvath K, Tarpley J, Savino JA, Gray R, Gulley J, Arnold T, Wolfe K, Risucci 
DA. Multi-institutional validation of a web-based core competency assessment system. JSurg Educ. 2007; 64: 390-394. 


2 . 




Standardized performance-based 
evaluations 


■ Attempt to model a real-world expert 
evaluation while eliminating or controlling 
variables 

■ Designed to assess similar attributes, behaviors, 
and competencies 

■ Examples: 

■ U.S. Medical Licensing Exam Step 2 Clinical Skills 

■ Primum Clinical Case Simulation ® 

■ OSCE for board certification in Family Medicine 




Standardized performance-based 
evaluations 


■ Limitations 

■ Behaviors and competencies hard to define 

■ Useful to assess demonstrable skills 

■ Poor generalizability beyond specific skills tested 

■ The broader the inference based on outcomes, the less valid 

■ Time and cost intensive 

■ Reliability and validity evidence in medical 
education literature moderate, but format 
widely used 

Cuddy MM, et al. Assessing the validity of the USMLE step 2 clinical knowledge examination through an evaluation of its 
clinical relevance. Acad Med. 2004; 10: S43-S45. 

KreiterCD, Bergus G. The validity of performance-based measures of clinical reasoning and alternative approaches. Med 
Educ. 2009; 43: 320-325. 

Jefferies A, et al. Using an objective structured clinical examination (OSCE) to assess multiple physician competencies in 
postgraduate training. MedTeach. 2007;29:183-191. 




Applications to training for 
exploration spaceflight 




What are we going to teach? 


■ Subcategories of training 

■ Medical knowledge 

■ Medical procedures and skills 
■ Dental procedures and skills 

■ Use of telemedicine 

■ Just-in-time training 

■ Hardware use 

■ Behavioral health 




Current medical training 


■ 2 CMOs from among the crewmembers for each 
International Space Station (ISS) mission 

■ No previous medical background 

■ Receive 40 hours of lecture and training 

■ Medical diagnostics (4 hours) 

■ Medical therapeutics (5.5 hours) 

■ CPR and ACLS (10.5 hours) 

■ Optional clinical component 


1. Hamilton D, Smart K, Melton S, Polk JD, Johnson-Throop K. Autonomous medical care for exploration class space missions. J 
Trauma. 2008; 64: S354-63. 

2. CMOTraining Flow, Wyle Laboratories, April 2010. 




Medical training for exploration 
spaceflight 


■ Must adapt the current training paradigms for 
the limitations of exploration missions 

■ Inability to return to Earth for evacuation 

■ Communication delays and blackouts 

■ Resource constraints 

■ Time 

■ Volume, mass, weight, power 

■ Cost 

■ As yet unperceived risks inherent in extended 
duration missions beyond low Earth orbit 


2 . 


Barsten K, Watkins SD, Otto CA. "Telemedicine workshop summary report," NASA TM 2012-xxxxxx, 2012. 

Risin D. "Risk of inability to adequately treat an ill or injured crewmember," in Human Health and Performance Risks of 
Space Exploration Missions . Ed. McPheeJC and Charles JB. NASA; 2009: 239-252. 




Medical procedures and skills 
training 



■ Minor procedures 
currently incorporated 
into CMO training 

■ Ongoing investigations 
into how to best 
perform procedures in 
microgravity 




Medical procedures and skills 
training for exploration spaceflight 


Define a list of procedures and skills necessary for 
each condition on the SMEMCL 
Prioritize procedures and skills based on urgency 
Driority of the condition 

Determine level of experience necessary for each 
condition 


1. Recognize and describe symptoms 

2. Make initial diagnosis 

3. Implement initial treatment intervention 

4. Treat to stabilization 

■ Determine how to optimize training for each 
procedure or skill given its priority 




Dental training 


■ ExMC gap 4.11: Limited dental care capabilities 

■ Dental working group held Mar 2012 

■ Participants were JSC /Wyle personnel, private dental 
practitioners, and an Army special forces dentist 

■ Addressed the following: 

■ Specific dental skills identified and prioritized 

■ Terrestrial training standards reviewed with emphasis on 
analogous environments (e.g., special forces medics) 

■ Areas for improvement in current crew training identified 
(e.g., hands-on experience in a high volume setting) 

■ Serves as a model for other SME panels 


Menon A, Barsten K, Watkins SD. "Dental working group meeting summary report," NASA TM 20i2-xxxxxx, 2012. 




Telemedicine training 


■ Telemedicine Workshop hosted by ExMC in 

2011 

■ Participants were internal personnel, external 
experts in telemedicine, and users of telemedicine 

■ Goal: Create a medical operational concept for a 
near-Earth asteroid mission 


Barsten K, Watkins SD, Otto CA. "Telemedicine workshop summary report," NASA TM 20i2-xxxxxx, 2012. 




Just-in-time (JIT) training 


■ Current use of "just-in-time training" 

■ CMOs trained by computer-based tutorials with real- 
time telemedicine guidance at point of care 

■ Used for ultrasonographic investigations on ISS with 
good results 

■ Need to develop a database of resources usable 
without real-time ground support 

■ Design optimized training programs for 
instruction in and practice with how to use 
resources 


1. CermackM. Monitoring and telemedicine support in remote environments and in human space flight. BrJ Anaesth. 2006; 
97: 101-14. 

2. von Lubitz DKJE. "Medical readiness training -just in time." Medical ReadinessTrainingTeam; U.S. Navy Advisory Team, 
University of Michigan. Retrieved from http://www.med-smart.org/downloads/NAVY_MEDICINE.pdf. 






Training in hardware use 



■ Current CMO training 
includes instruction on ISS 
medical hardware 

■ Need to engineer new 
hardware and software to 
address unique needs of 
exploration missions 

■ Need to design optimized 
training programs for new 
technology 




Behavioral health training 


■ Behavioral health of crew critical to mission 
success 

■ Crew medical officer must be able to 
diagnose and treat 

■ Best way to train yet to be elucidated 

■ Analysis of training for personnel in analogous 
remote and high-stress environments 




Prevention of knowledge and skill 
decay overtime 


■ Extensive studies in literature pertaininq to 
ACLS 

■ Rarity of events and infrequent or inadequate 
training are contributing factors 

■ Initial overtraining is a potential countermeasure 

■ Will need to be individualized 

■ Will need to be adapted to the timeline and 
the time constraints of the mission 


1. Barsten K, Watkins SD, Otto CA. "Telemedicine workshop summary report," NASA TM 20i2-xxxxxx, 2012. 

2. von Lubitz DKJE. "Medical readiness training -just in time." Medical ReadinessTrainingTeam; U.S. Navy Advisory Team, 
University of Michigan. Retrieved from http://www.med-smart.org/downloads/NAVY_MEDICINE.pdf. 




Key lessons learned 


■ There are no gold standards for training 

■ Must balance validity and reliability with 
feasibility and practicality 

■ Medical training optimization is an ongoing 
area of current research in analogous 
environments 

■ Any training procedures will require flexibility 
and constant adaptation to rapidly changing 
science of medicine 




Areas for future investigation 


■ Prioritize skills by the criticality and urgency 
of the associated condition 

■ Define to what experience level each skill 
must be taught 




Questions? 









References 


Baisden DL, Beven GE, Campbell MR, Charles JB, Dervay JP, Foster E, Gray GW, Hamilton 
DR, Holland DA, Jennings RT, Johnston SL, Jones JA, Kerwin JP, Locke J, Polk JD, Scarpa PJ, 

Sipes W, Stepanek J, Webb JT, Ad Hoc Committee of Members of the Space Medicine 
Association, Society of NASA Flight Surgeons. Human health and performance for longduration 
space flight. Aviat Space Environ Med. 2008 ; 79: 629-35. 

Barsten K, Watkins SD, Otto CA. "Telemedicine workshop summary report," NASA TM 2012-xxxxxx, 2012. 
Borsdoom D, Mellenbergh GJ. The concept of validity. Psychol Rev. 2004; 111: 1061-1071. 

Cermack M. Monitoring and telemedicine support in remote environments and in human space flight. BrJ 
Anaesth. 2006; 97: 101-14. 

CMO Training Flow, Wyle Laboratories, April 2010. 

Colliver JA, Conlee MJ, Verhulst SJ. From test validity to construct validity... and back? Med Educ. 2012; 46: 
366-371. 

CrossleyJ, Humphries G, Jolly B. Assessing health professionals. Med Educ. 2002;36:800-804. 

Cuddy MM, Dillon GF, Clauser BE, Holtzman KZ, Margolis MJ, McEllhenney SM, Swanson DB. Assessing the 
validity of the USMLE step 2 clinical knowledge examination through an evaluation of its clinical relevance. 
Acad Med. 2004; 10: S43-S45. 

Davis JR. Medical issues for a mission to Mars. Aviat Space Environ Med. 1999;70:162-68. 

Downing SM. Reliability: on the reproducibility of assessment data. Med Educ. 2004;38:1006-1012. 

"ExMC 3.01: We do not know the optimal training methods for in-flight medical conditions identified on the 
Space Medicine Exploration Medical Condition List taking into account the Crew Medical Officer's clinical 
background," Human Research Roadmap, NASA. Retrieved from 
http://humanresearchroadmap.nasa.gov/Gaps/7h387. 




References 


"ExMC Gap Report 3.01," NASA Human Research Wiki. Retrieved from 
https://humanresearchwiki.jsc.nasa.gov/index.php?title=3.oi. 

Jefferies A, Simmons B, Tabak D, Mcllroy JH, Lee KS, Roukema H, Skidmore M. Using an objective 
structured clinical examination (OSCE) to assess multiple physician competencies in postgraduate training. 
MedTeach. 2007; 29: 183-191. 

Hamilton D ; Smart K ; Melton S, Polk JD, Johnson-Throop K. Autonomous medical care for exploration class 
space missions. JTrauma. 2008; 64: S354-63. 

Kreiter CD, Bergus G. The validity of performance-based measures of clinical reasoning and alternative 
approaches. MedEduc. 2009;43:320-325. 

Lurie SJ, Mooney CJ, Lyness JM. Commentary: pitfalls in assessment of competency-based educational 
objectives. Acad Med. 2011; 86: 412-414. 

Menon A, Barsten K, Watkins SD. "Dental working group meeting summary report," NASA TM 2012-xxxxxx, 
2012. 

Risin D. "Risk of inability to adequately treat an ill or injured crewmember," in Human Health and 
Performance Risks of Space Exploration Missions . Ed. McPhee JC and Charles JB. NASA; 2009: 239-252. 
Summers RL, Johnston SL, MarshburnTH, Williams DR. Emergencies in space. Ann Emerg Med. 2005; 46: 
177-84. 

Tabuenca A, Welling R, Sachdeva AK, Blair PG, Horvath K, Tarpley J, Savino JA, Gray R, Gulley J, Arnold T, 
Wolfe K, Risucci DA. Multi-institutional validation of a web-based core competency assessment system. J 
Surg Educ. 2007; 64: 390-394. 

von Lubitz DKJE. "Medical readiness training -just in time." Medical ReadinessTrainingTeam; U.S. Navy 
Advisory Team, University of Michigan. Retrieved from http://www.med- 
smart.org/downloads/NAVY_MEDICINE.pdf. 




